双11全链路压测中通过Perf发现的一个SpringMVC 的性能问题

发表于 | 分类于 | 次

双11全链路压测中通过Perf发现的一个SpringMVC 的性能问题

在最近的全链路压测中TPS不够理想，然后通过perf 工具（perf record 采样， perf report 展示）看到(可以点击看大图)：

screenshot

再来看CPU消耗的火焰图：

screenshot

图中CPU的消耗占21%，不太正常。

可以看到Spring框架消耗了比较多的CPU，具体原因就是在Spring MVC中会大量使用到
@RequestMapping
@PathVariable
带来使用上的便利

业务方修改代码去掉spring中的methodMapping解析后的结果（性能提升了40%）：

screenshot.png

图中核心业务逻辑能抢到的cpu是21%（之前是15%）。spring methodMapping相关的也在火焰图中找不到了

Spring收到请求URL后要取出请求变量和做业务运算，具体代码(对照第一个图的调用堆栈）：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
170	public RequestMappingInfo More ...getMatchingCondition(HttpServletRequest request) {
171		RequestMethodsRequestCondition methods = methodsCondition.getMatchingCondition(request);
172		ParamsRequestCondition params = paramsCondition.getMatchingCondition(request);
173		HeadersRequestCondition headers = headersCondition.getMatchingCondition(request);
174		ConsumesRequestCondition consumes = consumesCondition.getMatchingCondition(request);
175		ProducesRequestCondition produces = producesCondition.getMatchingCondition(request);
176
177		if (methods == null || params == null || headers == null || consumes == null || produces == null) {
178			return null;
179		}
180
181		PatternsRequestCondition patterns = patternsCondition.getMatchingCondition(request);
182		if (patterns == null) {
183			return null;
184		}
185
186		RequestConditionHolder custom = customConditionHolder.getMatchingCondition(request);
187		if (custom == null) {
188			return null;
189		}
190
191		return new RequestMappingInfo(patterns, methods, params, headers, consumes, produces, custom.getCondition());
192	}

doMatch 代码：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
96 
97 	protected boolean More ...doMatch(String pattern, String path, boolean fullMatch,
98 			Map<String, String> uriTemplateVariables) {
99 
100		if (path.startsWith(this.pathSeparator) != pattern.startsWith(this.pathSeparator)) {
101			return false;
102		}
103
104		String[] pattDirs = StringUtils.tokenizeToStringArray(pattern, this.pathSeparator, this.trimTokens, true);
105		String[] pathDirs = StringUtils.tokenizeToStringArray(path, this.pathSeparator, this.trimTokens, true);
106
107		int pattIdxStart = 0;
108		int pattIdxEnd = pattDirs.length - 1;
109		int pathIdxStart = 0;
110		int pathIdxEnd = pathDirs.length - 1;
111
112		// Match all elements up to the first **
113		while (pattIdxStart <= pattIdxEnd && pathIdxStart <= pathIdxEnd) {
114			String patDir = pattDirs[pattIdxStart];
115			if ("**".equals(patDir)) {
116				break;
117			}
118			if (!matchStrings(patDir, pathDirs[pathIdxStart], uriTemplateVariables)) {
119				return false;
120			}
121			pattIdxStart++;
122			pathIdxStart++;
123		}
124
125		if (pathIdxStart > pathIdxEnd) {
126			// Path is exhausted, only match if rest of pattern is * or **'s
127			if (pattIdxStart > pattIdxEnd) {
128				return (pattern.endsWith(this.pathSeparator) ? path.endsWith(this.pathSeparator) :
129						!path.endsWith(this.pathSeparator));
130			}
131			if (!fullMatch) {
132				return true;
133			}
134			if (pattIdxStart == pattIdxEnd && pattDirs[pattIdxStart].equals("*") && path.endsWith(this.pathSeparator)) {
135				return true;
136			}
137			for (int i = pattIdxStart; i <= pattIdxEnd; i++) {
138				if (!pattDirs[i].equals("**")) {
139					return false;
140				}
141			}
142			return true;
143		}
144		else if (pattIdxStart > pattIdxEnd) {
145			// String not exhausted, but pattern is. Failure.
146			return false;
147		}
148		else if (!fullMatch && "**".equals(pattDirs[pattIdxStart])) {
149			// Path start definitely matches due to "**" part in pattern.
150			return true;
151		}
152
153		// up to last '**'
154		while (pattIdxStart <= pattIdxEnd && pathIdxStart <= pathIdxEnd) {
155			String patDir = pattDirs[pattIdxEnd];
156			if (patDir.equals("**")) {
157				break;
158			}
159			if (!matchStrings(patDir, pathDirs[pathIdxEnd], uriTemplateVariables)) {
160				return false;
161			}
162			pattIdxEnd--;
163			pathIdxEnd--;
164		}
165		if (pathIdxStart > pathIdxEnd) {
166			// String is exhausted
167			for (int i = pattIdxStart; i <= pattIdxEnd; i++) {
168				if (!pattDirs[i].equals("**")) {
169					return false;
170				}
171			}
172			return true;
173		}
174
175		while (pattIdxStart != pattIdxEnd && pathIdxStart <= pathIdxEnd) {
176			int patIdxTmp = -1;
177			for (int i = pattIdxStart + 1; i <= pattIdxEnd; i++) {
178				if (pattDirs[i].equals("**")) {
179					patIdxTmp = i;
180					break;
181				}
182			}
183			if (patIdxTmp == pattIdxStart + 1) {
184				// '**/**' situation, so skip one
185				pattIdxStart++;
186				continue;
187			}
188			// Find the pattern between padIdxStart & padIdxTmp in str between
189			// strIdxStart & strIdxEnd
190			int patLength = (patIdxTmp - pattIdxStart - 1);
191			int strLength = (pathIdxEnd - pathIdxStart + 1);
192			int foundIdx = -1;
193
194			strLoop:
195			for (int i = 0; i <= strLength - patLength; i++) {
196				for (int j = 0; j < patLength; j++) {
197					String subPat = pattDirs[pattIdxStart + j + 1];
198					String subStr = pathDirs[pathIdxStart + i + j];
199					if (!matchStrings(subPat, subStr, uriTemplateVariables)) {
200						continue strLoop;
201					}
202				}
203				foundIdx = pathIdxStart + i;
204				break;
205			}
206
207			if (foundIdx == -1) {
208				return false;
209			}
210
211			pattIdxStart = patIdxTmp;
212			pathIdxStart = foundIdx + patLength;
213		}
214
215		for (int i = pattIdxStart; i <= pattIdxEnd; i++) {
216			if (!pattDirs[i].equals("**")) {
217				return false;
218			}
219		}
220
221		return true;
222	}

最后补一个找到瓶颈点后 Google到类似问题的文章，并给出了具体数据和解决方法：http://www.cnblogs.com/ucos/articles/5542012.html

以及这篇文章中给出的优化前后对比图：
screenshot